SherbyNet - /page/wikibook/entry.mu

Micron Document

<!DOCTYPE html>

<title>Data engineering</title>

</head>

<span id="openzim-page-title" class="mw-page-title-main"><span class="mw-page-title-main">Data engineering</span></span>

</h1>

</header>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hatnote{font-style:italic}.mw-parser-output div.hatnote{padding-left:1.6em;margin-bottom:0.5em}.mw-parser-output .hatnote i{font-style:normal}.mw-parser-output .hatnote+link+.hatnote{margin-top:-0.5em}@media print{body.ns-0 .mw-parser-output .hatnote{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div role="note" class="hatnote navigation-not-searchable">Not to be confused with <a href="Information_Engineering" class="mw-redirect" title="Information Engineering">Information Engineering</a>.</div>

<p>

<b>Data engineering</b> is a <a href="Software_engineering" title="Software engineering">software engineering</a> approach to the building of <a href="Data_system" title="Data system">data systems</a>, to enable the collection and usage of <a href="Data" title="Data">data</a>. This data is usually used to enable subsequent <a href="Data_analytics" class="mw-redirect" title="Data analytics">analysis</a> and <a href="Data_science" title="Data science">data science</a>, which often involves <a href="Machine_learning" title="Machine learning">machine learning</a>.<sup id="cite_ref-whatis1_1-0" class="reference"><a href="#cite_note-whatis1-1"><span class="cite-bracket">[</span>1<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-whatis2_2-0" class="reference"><a href="#cite_note-whatis2-2"><span class="cite-bracket">[</span>2<span class="cite-bracket">]</span></a></sup> Making the data usable usually involves substantial <a href="Computer" title="Computer">compute</a> and <a href="Computer_data_storage" title="Computer data storage">storage</a>, as well as <a href="Data_processing" title="Data processing">data processing</a>.

</p>

<div class="mw-heading mw-heading2"><h2 id="History">History</h2></div>

<p>Around the 1970s/1980s the term <i><a href="Information_engineering" title="Information engineering">information engineering</a> methodology</i> (IEM) was created to describe <a href="Database_design" title="Database design">database design</a> and the use of <a href="Software" title="Software">software</a> for data analysis and processing.<sup id="cite_ref-hist1_3-0" class="reference"><a href="#cite_note-hist1-3"><span class="cite-bracket">[</span>3<span class="cite-bracket">]</span></a></sup> These techniques were intended to be used by <a href="Database_administrator" title="Database administrator">database administrators</a> (DBAs) and by <a href="Systems_analyst" title="Systems analyst">systems analysts</a> based upon an understanding of the operational processing needs of organizations for the 1980s. In particular, these techniques were meant to help bridge the gap between strategic business planning and information systems. A key early contributor (often called the "father" of information engineering methodology) was the Australian <a href="Clive_Finkelstein" title="Clive Finkelstein">Clive Finkelstein</a>, who wrote several articles about it between 1976 and 1980, and also co-authored an influential Savant Institute report on it with James Martin.<sup id="cite_ref-4" class="reference"><a href="#cite_note-4"><span class="cite-bracket">[</span>4<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-5" class="reference"><a href="#cite_note-5"><span class="cite-bracket">[</span>5<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-6" class="reference"><a href="#cite_note-6"><span class="cite-bracket">[</span>6<span class="cite-bracket">]</span></a></sup> Over the next few years, Finkelstein continued work in a more business-driven direction, which was intended to address a rapidly changing business environment; Martin continued work in a more data processing-driven direction. From 1983 to 1987, Charles M. Richter, guided by Clive Finkelstein, played a significant role in revamping IEM as well as helping to design the IEM software product (user data), which helped automate IEM.

</p><p>In the early 2000s, the data and data tooling was generally held by the <a href="Information_technology" title="Information technology">information technology</a> (IT) teams in most companies.<sup id="cite_ref-hist2_7-0" class="reference"><a href="#cite_note-hist2-7"><span class="cite-bracket">[</span>7<span class="cite-bracket">]</span></a></sup> Other teams then used data for their work (e.g. reporting), and there was usually little overlap in data skillset between these parts of the business.

</p><p>In the early 2010s, with the rise of the <a href="Internet" title="Internet">internet</a>, the massive increase in data volumes, velocity, and variety led to the term <a href="Big_data" title="Big data">big data</a> to describe the data itself, and data-driven tech companies like <a href="Facebook" title="Facebook">Facebook</a> and <a href="Airbnb" title="Airbnb">Airbnb</a> started using the phrase <b> data engineer</b>.<sup id="cite_ref-hist1_3-1" class="reference"><a href="#cite_note-hist1-3"><span class="cite-bracket">[</span>3<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-hist2_7-1" class="reference"><a href="#cite_note-hist2-7"><span class="cite-bracket">[</span>7<span class="cite-bracket">]</span></a></sup> Due to the new scale of the data, major firms like <a href="Google" title="Google">Google</a>, Facebook, <a href="Amazon_(company)" title="Amazon (company)">Amazon</a>, <a href="Apple_Inc." title="Apple Inc.">Apple</a>, <a href="Microsoft" title="Microsoft">Microsoft</a>, and <a href="Netflix" title="Netflix">Netflix</a> started to move away from traditional <a href="Extract_transform_load" class="mw-redirect" title="Extract transform load">ETL</a> and storage techniques. They started creating <b>data engineering</b>, a type of <a href="Software_engineering" title="Software engineering">software engineering</a> focused on data, and in particular <a href="Data_infrastructure" title="Data infrastructure">infrastructure</a>, <a href="Data_warehouse" title="Data warehouse">warehousing</a>, <a href="Information_privacy" title="Information privacy">data protection</a>, <a href="Cybersecurity" class="mw-redirect" title="Cybersecurity">cybersecurity</a>, <a href="Data_mining" title="Data mining">mining</a>, <a href="Data_modelling" class="mw-redirect" title="Data modelling">modelling</a>, <a href="Data_processing" title="Data processing">processing</a>, and <a href="Metadata" title="Metadata">metadata</a> management.<sup id="cite_ref-hist1_3-2" class="reference"><a href="#cite_note-hist1-3"><span class="cite-bracket">[</span>3<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-hist2_7-2" class="reference"><a href="#cite_note-hist2-7"><span class="cite-bracket">[</span>7<span class="cite-bracket">]</span></a></sup> This change in approach was particularly focused on <a href="Cloud_computing" title="Cloud computing">cloud computing</a>.<sup id="cite_ref-hist2_7-3" class="reference"><a href="#cite_note-hist2-7"><span class="cite-bracket">[</span>7<span class="cite-bracket">]</span></a></sup> Data started to be handled and used by many parts of the business, such as <a href="Sales" title="Sales">sales</a> and <a href="Marketing" title="Marketing">marketing</a>, and not just IT.<sup id="cite_ref-hist2_7-4" class="reference"><a href="#cite_note-hist2-7"><span class="cite-bracket">[</span>7<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading2"><h2 id="Tools">Tools</h2></div>

<div class="mw-heading mw-heading3"><h3 id="Compute">Compute</h3></div>

<p>High-performance computing is critical for the processing and analysis of data. One particularly widespread approach to computing for data engineering is <a href="Dataflow_programming" title="Dataflow programming">dataflow programming</a>, in which the computation is represented as a <a href="Directed_graph" title="Directed graph">directed graph</a> (dataflow graph); nodes are the operations, and edges represent the flow of data.<sup id="cite_ref-sigops_8-0" class="reference"><a href="#cite_note-sigops-8"><span class="cite-bracket">[</span>8<span class="cite-bracket">]</span></a></sup> Popular implementations include <a href="Apache_Spark" title="Apache Spark">Apache Spark</a>, and the <a href="Deep_learning" title="Deep learning">deep learning</a> specific <a href="TensorFlow" title="TensorFlow">TensorFlow</a>.<sup id="cite_ref-sigops_8-1" class="reference"><a href="#cite_note-sigops-8"><span class="cite-bracket">[</span>8<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-sparkpaper_9-0" class="reference"><a href="#cite_note-sparkpaper-9"><span class="cite-bracket">[</span>9<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-tensorflow_paper_10-0" class="reference"><a href="#cite_note-tensorflow_paper-10"><span class="cite-bracket">[</span>10<span class="cite-bracket">]</span></a></sup> More recent implementations, such as Differential/Timely Dataflow, have used <a href="Incremental_computing" title="Incremental computing">incremental computing</a> for much more efficient data processing.<sup id="cite_ref-sigops_8-2" class="reference"><a href="#cite_note-sigops-8"><span class="cite-bracket">[</span>8<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-differential-paper_11-0" class="reference"><a href="#cite_note-differential-paper-11"><span class="cite-bracket">[</span>11<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-differential-github_12-0" class="reference"><a href="#cite_note-differential-github-12"><span class="cite-bracket">[</span>12<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading3"><h3 id="Storage">Storage</h3></div>

<p>Data is stored in a variety of ways, one of the key deciding factors is in how the data will be used.

Data engineers optimize data storage and processing systems to reduce costs. They use data compression, partitioning, and archiving.

</p>

<div class="mw-heading mw-heading4"><h4 id="Databases">Databases</h4></div>

<p>If the data is structured and some form of <a href="Online_transaction_processing" title="Online transaction processing">online transaction processing</a> is required, then <a href="Databases" class="mw-redirect" title="Databases">databases</a> are generally used.<sup id="cite_ref-mit_13-0" class="reference"><a href="#cite_note-mit-13"><span class="cite-bracket">[</span>13<span class="cite-bracket">]</span></a></sup> Originally mostly <a href="Relational_database" title="Relational database">relational databases</a> were used, with strong <a href="ACID" title="ACID">ACID</a> transaction correctness guarantees; most relational databases use <a href="SQL" title="SQL">SQL</a> for their queries. However, with the growth of data in the 2010s, <a href="NoSQL" title="NoSQL">NoSQL</a> databases have also become popular since they <a href="Horizontal_scaling" class="mw-redirect" title="Horizontal scaling">horizontally scaled</a> more easily than relational databases by giving up the ACID transaction guarantees, as well as reducing the <a href="Object-relational_impedance_mismatch" class="mw-redirect" title="Object-relational impedance mismatch">object-relational impedance mismatch</a>.<sup id="cite_ref-leavitt_14-0" class="reference"><a href="#cite_note-leavitt-14"><span class="cite-bracket">[</span>14<span class="cite-bracket">]</span></a></sup> More recently, <a href="NewSQL" title="NewSQL">NewSQL</a> databases — which attempt to allow horizontal scaling while retaining ACID guarantees — have become popular.<sup id="cite_ref-aslett2012_15-0" class="reference"><a href="#cite_note-aslett2012-15"><span class="cite-bracket">[</span>15<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-sigmodrecord_16-0" class="reference"><a href="#cite_note-sigmodrecord-16"><span class="cite-bracket">[</span>16<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-17" class="reference"><a href="#cite_note-17"><span class="cite-bracket">[</span>17<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-high_scalability_18-0" class="reference"><a href="#cite_note-high_scalability-18"><span class="cite-bracket">[</span>18<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading4"><h4 id="Data_warehouses">Data warehouses</h4></div>

<div role="note" class="hatnote navigation-not-searchable">Main article: <a href="Data_warehouse" title="Data warehouse">Data warehouse</a></div>

<p>If the data is structured and <a href="Online_analytical_processing" title="Online analytical processing">online analytical processing</a> is required (but not online transaction processing), then <a href="Data_warehouse" title="Data warehouse">data warehouses</a> are a main choice.<sup id="cite_ref-ibm1_19-0" class="reference"><a href="#cite_note-ibm1-19"><span class="cite-bracket">[</span>19<span class="cite-bracket">]</span></a></sup> They enable data analysis, mining, and <a href="Artificial_intelligence" title="Artificial intelligence">artificial intelligence</a> on a much larger scale than databases can allow,<sup id="cite_ref-ibm1_19-1" class="reference"><a href="#cite_note-ibm1-19"><span class="cite-bracket">[</span>19<span class="cite-bracket">]</span></a></sup> and indeed data often flow from databases into data warehouses.<sup id="cite_ref-aws1_20-0" class="reference"><a href="#cite_note-aws1-20"><span class="cite-bracket">[</span>20<span class="cite-bracket">]</span></a></sup> <a href="Business_analyst" title="Business analyst">Business analysts</a>, data engineers, and data scientists can access data warehouses using tools such as SQL or <a href="Business_intelligence" title="Business intelligence">business intelligence</a> software.<sup id="cite_ref-aws1_20-1" class="reference"><a href="#cite_note-aws1-20"><span class="cite-bracket">[</span>20<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading4"><h4 id="Data_lakes">Data lakes</h4></div>

<p>A <a href="Data_lake" title="Data lake">data lake</a> is a centralized repository for storing, processing, and securing large volumes of data. A data lake can contain <a href="Structured_data" class="mw-redirect" title="Structured data">structured data</a> from <a href="Relational_database" title="Relational database">relational databases</a>, <a href="Semi-structured_data" title="Semi-structured data">semi-structured data</a>, <a href="Unstructured_data" title="Unstructured data">unstructured data</a>, and <a href="Binary_data" title="Binary data">binary data</a>. A data lake can be created on premises or in a cloud-based environment using the services from <a href="Cloud_computing" title="Cloud computing">public cloud</a> vendors such as <a href="Amazon_(company)" title="Amazon (company)">Amazon</a>, <a href="Microsoft" title="Microsoft">Microsoft</a>, or <a href="Google" title="Google">Google</a>.

</p>

<div class="mw-heading mw-heading4"><h4 id="Files">Files</h4></div>

<p>If the data is less structured, then often they are just stored as <a href="Computer_file" title="Computer file">files</a>. There are several options:

</p>

<ul><li><a href="File_system" title="File system">File systems</a> represent data hierarchically in nested folders.<sup id="cite_ref-redhat1_21-0" class="reference"><a href="#cite_note-redhat1-21"><span class="cite-bracket">[</span>21<span class="cite-bracket">]</span></a></sup></li>

<li><a href="Block_storage" class="mw-redirect" title="Block storage">Block storage</a> splits data into regularly sized chunks;<sup id="cite_ref-redhat1_21-1" class="reference"><a href="#cite_note-redhat1-21"><span class="cite-bracket">[</span>21<span class="cite-bracket">]</span></a></sup> this often matches up with (virtual) <a href="Hard_drives" class="mw-redirect" title="Hard drives">hard drives</a> or <a href="Solid_state_drives" class="mw-redirect" title="Solid state drives">solid state drives</a>.</li>

<li><a href="Object_storage" title="Object storage">Object storage</a> manages data using <a href="Metadata" title="Metadata">metadata</a>;<sup id="cite_ref-redhat1_21-2" class="reference"><a href="#cite_note-redhat1-21"><span class="cite-bracket">[</span>21<span class="cite-bracket">]</span></a></sup> often each file is assigned a key such as a <a href="UUID" class="mw-redirect" title="UUID">UUID</a>.<sup id="cite_ref-s3_22-0" class="reference"><a href="#cite_note-s3-22"><span class="cite-bracket">[</span>22<span class="cite-bracket">]</span></a></sup></li></ul>

<div class="mw-heading mw-heading3"><h3 id="Management">Management</h3></div>

<p>The number and variety of different data processes and storage locations can become overwhelming for users. This inspired the usage of a <a href="Workflow_management_system" title="Workflow management system">workflow management system</a> (e.g. <a href="Apache_Airflow" title="Apache Airflow">Airflow</a>) to allow the data tasks to be specified, created, and monitored.<sup id="cite_ref-airflow_23-0" class="reference"><a href="#cite_note-airflow-23"><span class="cite-bracket">[</span>23<span class="cite-bracket">]</span></a></sup> The tasks are often specified as a <a href="Directed_acyclic_graph" title="Directed acyclic graph">directed acyclic graph (DAG)</a>.<sup id="cite_ref-airflow_23-1" class="reference"><a href="#cite_note-airflow-23"><span class="cite-bracket">[</span>23<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading2"><h2 id="Lifecycle">Lifecycle</h2></div>

<div class="mw-heading mw-heading3"><h3 id="Business_planning">Business planning</h3></div>

<p>Business objectives that executives set for what's to come are characterized in key business plans, with their more noteworthy definition in tactical business plans and implementation in operational business plans. Most businesses today recognize the fundamental need to grow a business plan that follows this strategy. It is often difficult to implement these plans because of the lack of transparency at the tactical and operational degrees of organizations. This kind of planning requires feedback to allow for early correction of problems that are due to miscommunication and misinterpretation of the business plan.

</p>

<div class="mw-heading mw-heading3"><h3 id="Systems_design">Systems design</h3></div>

<p>The design of data systems involves several components such as architecting data platforms, and designing data stores.<sup id="cite_ref-course_24-0" class="reference"><a href="#cite_note-course-24"><span class="cite-bracket">[</span>24<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-25" class="reference"><a href="#cite_note-25"><span class="cite-bracket">[</span>25<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading3"><h3 id="Data_modeling">Data modeling</h3></div>

<div role="note" class="hatnote navigation-not-searchable">Main article: <a href="Data_modelling" class="mw-redirect" title="Data modelling">Data modelling</a></div>

<p>This is the process of producing a <a href="Data_model" title="Data model">data model</a>, an <a href="Abstract_model" class="mw-redirect" title="Abstract model">abstract model</a> to describe the data and relationships between different parts of the data.<sup id="cite_ref-model1_26-0" class="reference"><a href="#cite_note-model1-26"><span class="cite-bracket">[</span>26<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading2"><h2 id="Roles">Roles</h2></div>

<div class="mw-heading mw-heading3"><h3 id="Data_engineer">Data engineer</h3></div>

<p>A <b> data engineer</b> is a type of software engineer who creates <a href="Big_data" title="Big data">big data</a> <a href="Extract%2C_transform%2C_load" title="Extract, transform, load">ETL</a> pipelines to manage the flow of data through the organization. This makes it possible to take huge amounts of data and translate it into <a href="Business_intelligence" title="Business intelligence">insights</a>.<sup id="cite_ref-27" class="reference"><a href="#cite_note-27"><span class="cite-bracket">[</span>27<span class="cite-bracket">]</span></a></sup> They are focused on the production readiness of data and things like formats, resilience, scaling, and security. Data engineers usually hail from a software engineering background and are proficient in programming languages like <a href="Java_(programming_language)" title="Java (programming language)">Java</a>, <a href="Python_(programming_language)" title="Python (programming language)">Python</a>, <a href="Scala_(programming_language)" title="Scala (programming language)">Scala</a>, and <a href="Rust_(programming_language)" title="Rust (programming language)">Rust</a>.<sup id="cite_ref-28" class="reference"><a href="#cite_note-28"><span class="cite-bracket">[</span>28<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-hist1_3-3" class="reference"><a href="#cite_note-hist1-3"><span class="cite-bracket">[</span>3<span class="cite-bracket">]</span></a></sup> They will be more familiar with databases, architecture, cloud computing, and <a href="Agile_software_development" title="Agile software development">Agile software development</a>.<sup id="cite_ref-hist1_3-4" class="reference"><a href="#cite_note-hist1-3"><span class="cite-bracket">[</span>3<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading3"><h3 id="Data_scientist">Data scientist</h3></div>

<div role="note" class="hatnote navigation-not-searchable">Main article: <a href="Data_science" title="Data science">Data science</a></div>

<p><b>Data scientists</b> are more focused on the analysis of the data, they will be more familiar with <a href="Mathematics" title="Mathematics">mathematics</a>, <a href="Algorithms" class="mw-redirect" title="Algorithms">algorithms</a>, <a href="Statistics" title="Statistics">statistics</a>, and <a href="Machine_learning" title="Machine learning">machine learning</a>.<sup id="cite_ref-hist1_3-5" class="reference"><a href="#cite_note-hist1-3"><span class="cite-bracket">[</span>3<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-29" class="reference"><a href="#cite_note-29"><span class="cite-bracket">[</span>29<span class="cite-bracket">]</span></a></sup>

</p>

<li><a href="Information_technology" title="Information technology">Information technology</a></li>

<li><a href="Software_engineering" title="Software engineering">Software engineering</a></li>

<li><a href="Computer_science" title="Computer science">Computer science</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist reflist-columns references-column-width reflist-columns-2">

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.educba.com/what-is-data-engineering/">"What is Data Engineering? | A Quick Glance of Data Engineering"</a>. <i>EDUCBA</i>. January 5, 2020<span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-whatis2-2"><span class="mw-cite-backlink"><b><a href="#cite_ref-whatis2_2-0">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.dremio.com/resources/guides/intro-data-engineering/">"Introduction to Data Engineering"</a>. <i>Dremio</i><span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-hist1-3"><span class="mw-cite-backlink">^ <a href="#cite_ref-hist1_3-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-hist1_3-1"><sup><i><b>b</b></i></sup></a> <a href="#cite_ref-hist1_3-2"><sup><i><b>c</b></i></sup></a> <a href="#cite_ref-hist1_3-3"><sup><i><b>d</b></i></sup></a> <a href="#cite_ref-hist1_3-4"><sup><i><b>e</b></i></sup></a> <a href="#cite_ref-hist1_3-5"><sup><i><b>f</b></i></sup></a></span> <span class="reference-text"><cite id="CITEREFBlack2020" class="citation web cs1">Black, Nathan (January 15, 2020). <a rel="nofollow" class="external text" href="https://quanthub.com/what-is-data-engineering/">"What is Data Engineering and Why Is It So Important?"</a>. <i>QuantHub</i><span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-4"><span class="mw-cite-backlink"><b><a href="#cite_ref-4">^</a></b></span> <span class="reference-text">"Information engineering," <a rel="nofollow" class="external text" href="https://books.google.com/books?id=U2Da-O9RAgIC&pg=PA29">part 3</a>, <a rel="nofollow" class="external text" href="https://books.google.com/books?id=aMrnCDJzb9MC&pg=RA1-PA1">part 4</a>, <a rel="nofollow" class="external text" href="https://books.google.com/books?id=Ux9iw6tMs6MC&pg=PA32">part 5</a>, <a rel="nofollow" class="external text" href="https://books.google.com/books?id=dPLZ7QidjbEC&pg=RA1-PA1">Part 6</a>" by Clive Finkelstein. In <i>Computerworld, In depths, appendix.</i> May 25 – June 15, 1981.</span>

</li>

<li id="cite_note-5"><span class="mw-cite-backlink"><b><a href="#cite_ref-5">^</a></b></span> <span class="reference-text">Christopher Allen, Simon Chatwin, Catherine Creary (2003). <i>Introduction to Relational Databases and SQL Programming.</i></span>

</li>

<li id="cite_note-6"><span class="mw-cite-backlink"><b><a href="#cite_ref-6">^</a></b></span> <span class="reference-text"><a href="Terry_Halpin" title="Terry Halpin">Terry Halpin</a>, <a href="Tony_Morgan_(computer_scientist)" title="Tony Morgan (computer scientist)">Tony Morgan</a> (2010). <i>Information Modeling and Relational Databases.</i> p. 343</span>

</li>

<li id="cite_note-hist2-7"><span class="mw-cite-backlink">^ <a href="#cite_ref-hist2_7-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-hist2_7-1"><sup><i><b>b</b></i></sup></a> <a href="#cite_ref-hist2_7-2"><sup><i><b>c</b></i></sup></a> <a href="#cite_ref-hist2_7-3"><sup><i><b>d</b></i></sup></a> <a href="#cite_ref-hist2_7-4"><sup><i><b>e</b></i></sup></a></span> <span class="reference-text"><cite id="CITEREFDodds" class="citation web cs1">Dodds, Eric. <a rel="nofollow" class="external text" href="https://www.rudderstack.com/blog/the-data-engineering-megatrend-a-brief-history">"The History of the Data Engineering and the Megatrends"</a>. <i>Rudderstack</i><span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-sigops-8"><span class="mw-cite-backlink">^ <a href="#cite_ref-sigops_8-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-sigops_8-1"><sup><i><b>b</b></i></sup></a> <a href="#cite_ref-sigops_8-2"><sup><i><b>c</b></i></sup></a></span> <span class="reference-text"><cite id="CITEREFSchwarzkopf2020" class="citation web cs1">Schwarzkopf, Malte (March 7, 2020). <a rel="nofollow" class="external text" href="https://www.sigops.org/2020/the-remarkable-utility-of-dataflow-computing/">"The Remarkable Utility of Dataflow Computing"</a>. <i>ACM SIGOPS</i><span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-sparkpaper-9"><span class="mw-cite-backlink"><b><a href="#cite_ref-sparkpaper_9-0">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://cs.stanford.edu/~matei/papers/2016/cacm_apache_spark.pdf">"sparkpaper"</a> <span class="cs1-format">(PDF)</span><span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-tensorflow_paper-10"><span class="mw-cite-backlink"><b><a href="#cite_ref-tensorflow_paper_10-0">^</a></b></span> <span class="reference-text"><cite id="CITEREFAbadiBarhamChenChen2016" class="citation web cs1">Abadi, Martin; Barham, Paul; Chen, Jianmin; Chen, Zhifeng; Davis, Andy; Dean, Jeffrey; Devin, Matthieu; Ghemawat, Sanjay; Irving, Geoffrey; Isard, Michael; Kudlur, Manjunath; Levenberg, Josh; Monga, Rajat; Moore, Sherry; Murray, Derek G.; Steiner, Benoit; Tucker, Paul; Vasudevan, Vijay; Warden, Pete; Wicke, Martin; Yu, Yuan; Zheng, Xiaoqiang (2016). <a rel="nofollow" class="external text" href="https://research.google/pubs/pub45381/">"TensorFlow: A system for large-scale machine learning"</a>. <i>12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)</i>. pp. <span class="nowrap">265–</span>283<span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-differential-paper-11"><span class="mw-cite-backlink"><b><a href="#cite_ref-differential-paper_11-0">^</a></b></span> <span class="reference-text"><cite id="CITEREFMcSherryMurrayIsaacsIsard2013" class="citation web cs1">McSherry, Frank; Murray, Derek; Isaacs, Rebecca; Isard, Michael (January 5, 2013). <a rel="nofollow" class="external text" href="https://www.microsoft.com/en-us/research/publication/differential-dataflow/">"Differential dataflow"</a>. <i><a href="Microsoft" title="Microsoft">Microsoft</a></i><span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-differential-github-12"><span class="mw-cite-backlink"><b><a href="#cite_ref-differential-github_12-0">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://github.com/TimelyDataflow/differential-dataflow">"Differential Dataflow"</a>. Timely Dataflow. July 30, 2022<span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-mit-13"><span class="mw-cite-backlink"><b><a href="#cite_ref-mit_13-0">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://ocw.mit.edu/courses/6-830-database-systems-fall-2010/pages/lecture-notes/">"Lecture Notes | Database Systems | Electrical Engineering and Computer Science | MIT OpenCourseWare"</a>. <i>ocw.mit.edu</i><span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-leavitt-14"><span class="mw-cite-backlink"><b><a href="#cite_ref-leavitt_14-0">^</a></b></span> <span class="reference-text"><cite id="CITEREFLeavitt2010" class="citation journal cs1">Leavitt, Neal (February 2010). "Will NoSQL Databases Live Up to Their Promise?". <i>Computer</i>. <b>43</b> (2): <span class="nowrap">12–</span>14. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FMC.2010.58">10.1109/MC.2010.58</a>.</cite></span>

</li>

<li id="cite_note-aslett2012-15"><span class="mw-cite-backlink"><b><a href="#cite_ref-aslett2012_15-0">^</a></b></span> <span class="reference-text"><cite id="CITEREFAslett2011" class="citation web cs1">Aslett, Matthew (2011). <a rel="nofollow" class="external text" href="http://cs.brown.edu/courses/cs227/archives/2012/papers/newsql/aslett-newsql.pdf">"How Will The Database Incumbents Respond To NoSQL And NewSQL?"</a> <span class="cs1-format">(PDF)</span>. 451 Group (published April 4, 2011)<span class="reference-accessdate">. Retrieved <span class="nowrap">February 22,</span> 2020</span>.</cite></span>

</li>

<li id="cite_note-sigmodrecord-16"><span class="mw-cite-backlink"><b><a href="#cite_ref-sigmodrecord_16-0">^</a></b></span> <span class="reference-text"><cite id="CITEREFPavloAslett2016" class="citation conference cs1">Pavlo, Andrew; Aslett, Matthew (2016). <a rel="nofollow" class="external text" href="https://db.cs.cmu.edu/papers/2016/pavlo-newsql-sigmodrec2016.pdf">"What's Really New with NewSQL?"</a> <span class="cs1-format">(PDF)</span>. <i>SIGMOD Record</i><span class="reference-accessdate">. Retrieved <span class="nowrap">February 22,</span> 2020</span>.</cite></span>

</li>

<li id="cite_note-17"><span class="mw-cite-backlink"><b><a href="#cite_ref-17">^</a></b></span> <span class="reference-text"><cite id="CITEREFStonebraker2011" class="citation web cs1">Stonebraker, Michael (June 16, 2011). <a rel="nofollow" class="external text" href="https://cacm.acm.org/blogs/blog-cacm/109710-new-sql-an-alternative-to-nosql-and-old-sql-for-new-oltp-apps/fulltext">"NewSQL: An Alternative to NoSQL and Old SQL for New OLTP Apps"</a>. Communications of the ACM Blog<span class="reference-accessdate">. Retrieved <span class="nowrap">February 22,</span> 2020</span>.</cite></span>

</li>

<li id="cite_note-high_scalability-18"><span class="mw-cite-backlink"><b><a href="#cite_ref-high_scalability_18-0">^</a></b></span> <span class="reference-text"><cite id="CITEREFHoff2012" class="citation web cs1">Hoff, Todd (September 24, 2012). <a rel="nofollow" class="external text" href="http://highscalability.com/blog/2012/9/24/google-spanners-most-surprising-revelation-nosql-is-out-and.html">"Google Spanner's Most Surprising Revelation: NoSQL is Out and NewSQL is In"</a><span class="reference-accessdate">. Retrieved <span class="nowrap">February 22,</span> 2020</span>.</cite></span>

</li>

<li id="cite_note-ibm1-19"><span class="mw-cite-backlink">^ <a href="#cite_ref-ibm1_19-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-ibm1_19-1"><sup><i><b>b</b></i></sup></a></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.ibm.com/cloud/learn/data-warehouse">"What is a Data Warehouse?"</a>. <i>www.ibm.com</i><span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-aws1-20"><span class="mw-cite-backlink">^ <a href="#cite_ref-aws1_20-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-aws1_20-1"><sup><i><b>b</b></i></sup></a></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://aws.amazon.com/data-warehouse/">"What is a Data Warehouse? | Key Concepts | Amazon Web Services"</a>. <i>Amazon Web Services, Inc</i><span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-redhat1-21"><span class="mw-cite-backlink">^ <a href="#cite_ref-redhat1_21-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-redhat1_21-1"><sup><i><b>b</b></i></sup></a> <a href="#cite_ref-redhat1_21-2"><sup><i><b>c</b></i></sup></a></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.redhat.com/en/topics/data-storage/file-block-object-storage">"File storage, block storage, or object storage?"</a>. <i>www.redhat.com</i><span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-s3-22"><span class="mw-cite-backlink"><b><a href="#cite_ref-s3_22-0">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://aws.amazon.com/s3">"Cloud Object Storage – Amazon S3 – Amazon Web Services"</a>. <i>Amazon Web Services, Inc</i><span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-airflow-23"><span class="mw-cite-backlink">^ <a href="#cite_ref-airflow_23-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-airflow_23-1"><sup><i><b>b</b></i></sup></a></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://airflow.apache.org/">"Home"</a>. <i>Apache Airflow</i><span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-course-24"><span class="mw-cite-backlink"><b><a href="#cite_ref-course_24-0">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.coursera.org/learn/introduction-to-data-engineering">"Introduction to Data Engineering"</a>. <i>Coursera</i><span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-25"><span class="mw-cite-backlink"><b><a href="#cite_ref-25">^</a></b></span> <span class="reference-text"><cite id="CITEREFFinkelstein" class="citation book cs1">Finkelstein, Clive. <i>What are The Phases of Information Engineering</i>.</cite></span>

</li>

<li id="cite_note-model1-26"><span class="mw-cite-backlink"><b><a href="#cite_ref-model1_26-0">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.simplilearn.com/what-is-data-modeling-article">"What is Data Modelling? Overview, Basic Concepts, and Types in Detail"</a>. <i>Simplilearn.com</i>. June 15, 2021<span class="reference-accessdate">. Retrieved <span class="nowrap">July 31,</span> 2022</span>.</cite></span>

</li>

<li id="cite_note-27"><span class="mw-cite-backlink"><b><a href="#cite_ref-27">^</a></b></span> <span class="reference-text"><cite id="CITEREFTamirMillerGagliardi2015" class="citation report cs1">Tamir, Mike; Miller, Steven; Gagliardi, Alessandro (December 11, 2015). The Data Engineer (Report). <a href="SSRN_(identifier)" class="mw-redirect" title="SSRN (identifier)">SSRN</a> <a rel="nofollow" class="external text" href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2762013">2762013</a>.</cite></span>

</li>

<li id="cite_note-28"><span class="mw-cite-backlink"><b><a href="#cite_ref-28">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.springboard.com/blog/data-engineer-vs-data-scientist/">"Data Engineer vs. Data Scientist"</a>. <i>Springboard Blog</i>. February 7, 2019<span class="reference-accessdate">. Retrieved <span class="nowrap">March 14,</span> 2021</span>.</cite></span>

</li>

<li id="cite_note-29"><span class="mw-cite-backlink"><b><a href="#cite_ref-29">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.edureka.co/blog/what-is-data-science/">"What is Data Science and Why it's Important"</a>. Edureka. January 5, 2017.</cite></span>

</li>

</ol></div>

<div class="mw-heading mw-heading2"><h2 id="Further_reading">Further reading</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .refbegin{margin-bottom:0.5em}.mw-parser-output .refbegin-hanging-indents>ul{margin-left:0}.mw-parser-output .refbegin-hanging-indents>ul>li{margin-left:0;padding-left:3.2em;text-indent:-3.2em}.mw-parser-output .refbegin-hanging-indents ul,.mw-parser-output .refbegin-hanging-indents ul li{list-style:none}@media(max-width:720px){.mw-parser-output .refbegin-hanging-indents>ul>li{padding-left:1.6em;text-indent:-1.6em}}.mw-parser-output .refbegin-columns{margin-top:0.3em}.mw-parser-output .refbegin-columns ul{margin-top:0}.mw-parser-output .refbegin-columns li{page-break-inside:avoid;break-inside:avoid-column}@media screen{.mw-parser-output .refbegin{font-size:90%}}

/* end https://en.wikipedia.org/ */

</style><div class="refbegin refbegin-columns references-column-width" style="column-width: 30em">

<ul><li><cite id="CITEREFHares1992" class="citation book cs1">Hares, John S. (1992). <i>Information Engineering for the Advanced Practitioner</i>. Wiley. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-471-92810-2</bdi>.</cite></li>

<li><cite id="CITEREFFinkelstein1989" class="citation book cs1">Finkelstein, Clive (1989). <i>An Introduction to Information Engineering: From Strategic Planning to Information Systems</i>. Addison-Wesley. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-201-41654-1</bdi>.</cite></li>

<li><cite id="CITEREFFinkelstein1992" class="citation book cs1">Finkelstein, Clive (1992). <i>Information Engineering: Strategic Systems Development</i>. Addison-Wesley. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-201-50988-5</bdi>.</cite></li>

<li>Ian Macdonald (1986). "Information engineering". in: <i>Information Systems Design Methodologies</i>. T.W. Olle et al. (ed.). North-Holland.</li>

<li>Ian Macdonald (1988). "Automating the Information engineering methodology with the Information Engineering Facility". In: <i>Computerized Assistance during the Information Systems Life Cycle</i>. <a href="T.W._Olle" class="mw-redirect" title="T.W. Olle">T.W. Olle</a> et al. (ed.). North-Holland.</li>

<li><a href="James_Martin_(author)" title="James Martin (author)">James Martin</a> and <a href="Clive_Finkelstein" title="Clive Finkelstein">Clive Finkelstein</a>. (1981). <i>Information engineering</i>. Technical Report (2 volumes), Savant Institute, Carnforth, Lancs, UK.</li>

<li>James Martin (1989). <i>Information engineering</i>. (3 volumes), Prentice-Hall Inc.</li>

<li><cite id="CITEREFFinkelstein2006" class="citation book cs1">Finkelstein, Clive (2006). <i>Enterprise Architecture for Integration: Rapid Delivery Methods and Technologies</i>. Artech House. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-1-58053-713-1</bdi>.</cite></li>

<li><cite id="CITEREFReisHousley2022" class="citation book cs1">Reis, Joe; Housley, Matt (2022). <i>Fundamentals of Data Engineering</i>. O'Reilly Media. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-1-0981-0827-4</bdi>.</cite></li></ul>

</div>

<div class="mw-heading mw-heading2"><h2 id="External_links">External links</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .side-box{margin:4px 0;box-sizing:border-box;border:1px solid #aaa;font-size:88%;line-height:1.25em;background-color:var(--background-color-interactive-subtle,#f8f9fa);display:flow-root}.mw-parser-output .infobox .side-box{font-size:100%}.mw-parser-output .side-box-abovebelow,.mw-parser-output .side-box-text{padding:0.25em 0.9em}.mw-parser-output .side-box-image{padding:2px 0 2px 0.9em;text-align:center}.mw-parser-output .side-box-imageright{padding:2px 0.9em 2px 0;text-align:center}@media(min-width:500px){.mw-parser-output .side-box-flex{display:flex;align-items:center}.mw-parser-output .side-box-text{flex:1;min-width:0}}@media(min-width:720px){.mw-parser-output .side-box{width:238px}.mw-parser-output .side-box-right{clear:right;float:right;margin-left:1em}.mw-parser-output .side-box-left{margin-right:1em}}

/* end https://en.wikipedia.org/ */

</style><style data-mw-deduplicate="TemplateStyles:r1237033735">

/* start https://en.wikipedia.org/ */

@media print{body.ns-0 .mw-parser-output .sistersitebox{display:none!important}}@media screen{html.skin-theme-clientpref-night .mw-parser-output .sistersitebox img[src*="Wiktionary-logo-en-v2.svg"]{background-color:white}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .sistersitebox img[src*="Wiktionary-logo-en-v2.svg"]{background-color:white}}

/* end https://en.wikipedia.org/ */

</style><div class="side-box side-box-right sistersitebox"><style data-mw-deduplicate="TemplateStyles:r1126788409">

/* start https://en.wikipedia.org/ */

.mw-parser-output .plainlist ol,.mw-parser-output .plainlist ul{line-height:inherit;list-style:none;margin:0;padding:0}.mw-parser-output .plainlist ol li,.mw-parser-output .plainlist ul li{margin-bottom:0}

/* end https://en.wikipedia.org/ */

</style>

<div class="side-box-text plainlist">Wikimedia Commons has media related to <span style="font-weight: bold; font-style: italic;"><a href="https://commons.wikimedia.org/wiki/Category:_Information_Engineering" class="extiw external" title="commons:Category: Information Engineering">Information Engineering</a></span>.</div></div>

</div>

<ul><li><a rel="nofollow" class="external text" href="http://www.informatik.uni-bremen.de/uniform/gdpa/methods/m-iem.htm">The Complex Method IEM</a> <a rel="nofollow" class="external text" href="https://web.archive.org/web/20190720070308/http://www.informatik.uni-bremen.de/uniform/gdpa/methods/m-iem.htm">Archived</a> July 20, 2019, at the <a href="Wayback_Machine" title="Wayback Machine">Wayback Machine</a></li>

<li><a rel="nofollow" class="external text" href="https://web.archive.org/web/20060215222446/http://sysdev.ucdavis.edu/WEBADM/document/rad-archapproach.htm">Rapid Application Development</a></li>

<li><a rel="nofollow" class="external text" href="http://www.ies.aust.com">Enterprise Engineering and Rapid Delivery of Enterprise Architecture</a></li></ul>

<p><br>

</p>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hlist dl,.mw-parser-output .hlist ol,.mw-parser-output .hlist ul{margin:0;padding:0}.mw-parser-output .hlist dd,.mw-parser-output .hlist dt,.mw-parser-output .hlist li{margin:0;display:inline}.mw-parser-output .hlist.inline,.mw-parser-output .hlist.inline dl,.mw-parser-output .hlist.inline ol,.mw-parser-output .hlist.inline ul,.mw-parser-output .hlist dl dl,.mw-parser-output .hlist dl ol,.mw-parser-output .hlist dl ul,.mw-parser-output .hlist ol dl,.mw-parser-output .hlist ol ol,.mw-parser-output .hlist ol ul,.mw-parser-output .hlist ul dl,.mw-parser-output .hlist ul ol,.mw-parser-output .hlist ul ul{display:inline}.mw-parser-output .hlist .mw-empty-li{display:none}.mw-parser-output .hlist dt::after{content:": "}.mw-parser-output .hlist dd::after,.mw-parser-output .hlist li::after{content:" · ";font-weight:bold}.mw-parser-output .hlist dd:last-child::after,.mw-parser-output .hlist dt:last-child::after,.mw-parser-output .hlist li:last-child::after{content:none}.mw-parser-output .hlist dd dd:first-child::before,.mw-parser-output .hlist dd dt:first-child::before,.mw-parser-output .hlist dd li:first-child::before,.mw-parser-output .hlist dt dd:first-child::before,.mw-parser-output .hlist dt dt:first-child::before,.mw-parser-output .hlist dt li:first-child::before,.mw-parser-output .hlist li dd:first-child::before,.mw-parser-output .hlist li dt:first-child::before,.mw-parser-output .hlist li li:first-child::before{content:" (";font-weight:normal}.mw-parser-output .hlist dd dd:last-child::after,.mw-parser-output .hlist dd dt:last-child::after,.mw-parser-output .hlist dd li:last-child::after,.mw-parser-output .hlist dt dd:last-child::after,.mw-parser-output .hlist dt dt:last-child::after,.mw-parser-output .hlist dt li:last-child::after,.mw-parser-output .hlist li dd:last-child::after,.mw-parser-output .hlist li dt:last-child::after,.mw-parser-output .hlist li li:last-child::after{content:")";font-weight:normal}.mw-parser-output .hlist ol{counter-reset:listitem}.mw-parser-output .hlist ol>li{counter-increment:listitem}.mw-parser-output .hlist ol>li::before{content:" "counter(listitem)"\a0 "}.mw-parser-output .hlist dd ol>li:first-child::before,.mw-parser-output .hlist dt ol>li:first-child::before,.mw-parser-output .hlist li ol>li:first-child::before{content:" ("counter(listitem)"\a0 "}

/* end https://en.wikipedia.org/ */

</style><style data-mw-deduplicate="TemplateStyles:r1236075235">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbox{box-sizing:border-box;border:1px solid #a2a9b1;width:100%;clear:both;font-size:88%;text-align:center;padding:1px;margin:1em auto 0}.mw-parser-output .navbox .navbox{margin-top:0}.mw-parser-output .navbox+.navbox,.mw-parser-output .navbox+.navbox-styles+.navbox{margin-top:-1px}.mw-parser-output .navbox-inner,.mw-parser-output .navbox-subgroup{width:100%}.mw-parser-output .navbox-group,.mw-parser-output .navbox-title,.mw-parser-output .navbox-abovebelow{padding:0.25em 1em;line-height:1.5em;text-align:center}.mw-parser-output .navbox-group{white-space:nowrap;text-align:right}.mw-parser-output .navbox,.mw-parser-output .navbox-subgroup{background-color:#fdfdfd}.mw-parser-output .navbox-list{line-height:1.5em;border-color:#fdfdfd}.mw-parser-output .navbox-list-with-group{text-align:left;border-left-width:2px;border-left-style:solid}.mw-parser-output tr+tr>.navbox-abovebelow,.mw-parser-output tr+tr>.navbox-group,.mw-parser-output tr+tr>.navbox-image,.mw-parser-output tr+tr>.navbox-list{border-top:2px solid #fdfdfd}.mw-parser-output .navbox-title{background-color:#ccf}.mw-parser-output .navbox-abovebelow,.mw-parser-output .navbox-group,.mw-parser-output .navbox-subgroup .navbox-title{background-color:#ddf}.mw-parser-output .navbox-subgroup .navbox-group,.mw-parser-output .navbox-subgroup .navbox-abovebelow{background-color:#e6e6ff}.mw-parser-output .navbox-even{background-color:#f7f7f7}.mw-parser-output .navbox-odd{background-color:transparent}.mw-parser-output .navbox .hlist td dl,.mw-parser-output .navbox .hlist td ol,.mw-parser-output .navbox .hlist td ul,.mw-parser-output .navbox td.hlist dl,.mw-parser-output .navbox td.hlist ol,.mw-parser-output .navbox td.hlist ul{padding:0.125em 0}.mw-parser-output .navbox .navbar{display:block;font-size:100%}.mw-parser-output .navbox-title .navbar{float:left;text-align:left;margin-right:0.5em}body.skin--responsive .mw-parser-output .navbox-image img{max-width:none!important}@media print{body.ns-0 .mw-parser-output .navbox{display:none!important}}

/* end https://en.wikipedia.org/ */

</style></div><div role="navigation" class="navbox" aria-labelledby="Engineering683" style="padding:3px"><table class="nowraplinks hlist mw-collapsible autocollapse navbox-inner" style="border-spacing:0;background:transparent;color:inherit"><tbody><tr><th scope="col" class="navbox-title" colspan="3"><style data-mw-deduplicate="TemplateStyles:r1239400231">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbar{display:inline;font-size:88%;font-weight:normal}.mw-parser-output .navbar-collapse{float:left;text-align:left}.mw-parser-output .navbar-boxtext{word-spacing:0}.mw-parser-output .navbar ul{display:inline-block;white-space:nowrap;line-height:inherit}.mw-parser-output .navbar-brackets::before{margin-right:-0.125em;content:"[ "}.mw-parser-output .navbar-brackets::after{margin-left:-0.125em;content:" ]"}.mw-parser-output .navbar li{word-spacing:-0.125em}.mw-parser-output .navbar a>span,.mw-parser-output .navbar a>abbr{text-decoration:inherit}.mw-parser-output .navbar-mini abbr{font-variant:small-caps;border-bottom:none;text-decoration:none;cursor:inherit}.mw-parser-output .navbar-ct-full{font-size:114%;margin:0 7em}.mw-parser-output .navbar-ct-mini{font-size:114%;margin:0 4em}html.skin-theme-clientpref-night .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}@media(prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}}@media print{.mw-parser-output .navbar{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div id="Engineering683" style="font-size:114%;margin:0 4em"><a href="Engineering" title="Engineering">Engineering</a></div></th></tr><tr><td class="navbox-abovebelow" colspan="3"><div>

<ul><li><a href="History_of_engineering" title="History of engineering">History</a></li>

<li><a href="Outline_of_engineering" title="Outline of engineering">Outline</a></li>

<li><a href="List_of_engineering_branches" title="List of engineering branches">List of engineering branches</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Specialties<br>and<br><a href="Interdisciplinarity" title="Interdisciplinarity">interdisciplinarity</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Civil_engineering" title="Civil engineering">Civil</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Architectural_engineering" title="Architectural engineering">Architectural</a></li>

<li><a href="Coastal_engineering" title="Coastal engineering">Coastal</a></li>

<li><a href="Construction_engineering" title="Construction engineering">Construction</a></li>

<li><a href="Earthquake_engineering" title="Earthquake engineering">Earthquake</a></li>

<li><a href="Ecological_engineering" title="Ecological engineering">Ecological</a></li>

<li><a href="Environmental_engineering" title="Environmental engineering">Environmental</a>

<ul><li><a href="Sanitary_engineering" title="Sanitary engineering">Sanitary</a></li></ul></li>

<li><a href="Geological_engineering" title="Geological engineering">Geological</a></li>

<li><a href="Geotechnical_engineering" title="Geotechnical engineering">Geotechnical</a></li>

<li><a href="Hydraulic_engineering" title="Hydraulic engineering">Hydraulic</a></li>

<li><a href="Mining_engineering" title="Mining engineering">Mining</a></li>

<li><a href="Municipal_or_urban_engineering" title="Municipal or urban engineering">Municipal/urban</a></li>

<li><a href="Offshore_engineering" class="mw-redirect" title="Offshore engineering">Offshore</a></li>

<li><a href="River_engineering" title="River engineering">River</a></li>

<li><a href="Structural_engineering" title="Structural engineering">Structural</a></li>

<li><a href="Transportation_engineering" title="Transportation engineering">Transportation</a>

<ul><li><a href="Traffic_engineering_(transportation)" title="Traffic engineering (transportation)">Traffic</a></li>

<li><a href="Railway_engineering" title="Railway engineering">Railway</a></li></ul></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Mechanical_engineering" title="Mechanical engineering">Mechanical</a></th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Acoustical_engineering" title="Acoustical engineering">Acoustic</a></li>

<li><a href="Aerospace_engineering" title="Aerospace engineering">Aerospace</a></li>

<li><a href="Automotive_engineering" title="Automotive engineering">Automotive</a></li>

<li><a href="Biomechanical_engineering" title="Biomechanical engineering">Biomechanical</a></li>

<li><a href="Energy_engineering" title="Energy engineering">Energy</a></li>

<li><a href="Manufacturing_engineering" title="Manufacturing engineering">Manufacturing</a></li>

<li><a href="Marine_engineering" title="Marine engineering">Marine</a></li>

<li><a href="Naval_architecture" title="Naval architecture">Naval architecture</a></li>

<li><a href="Railway_engineering" title="Railway engineering">Railway</a></li>

<li><a href="Sports_engineering" title="Sports engineering">Sports</a></li>

<li><a href="Thermal_engineering" title="Thermal engineering">Thermal</a></li>

<li><a href="Tribology" title="Tribology">Tribology</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Electrical_engineering" title="Electrical engineering">Electrical</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Broadcast_engineering" title="Broadcast engineering">Broadcast</a>

<ul><li><a href="Outline_of_computer_engineering" title="Outline of computer engineering">outline</a></li></ul></li>

<li><a href="Control_engineering" title="Control engineering">Control</a></li>

<li><a href="Electromechanics" title="Electromechanics">Electromechanics</a></li>

<li><a href="Electronic_engineering" title="Electronic engineering">Electronics</a></li>

<li><a href="Microwave_engineering" title="Microwave engineering">Microwaves</a></li>

<li><a href="Optical_engineering" title="Optical engineering">Optical</a></li>

<li><a href="Power_engineering" title="Power engineering">Power</a></li>

<li><a href="Radio-frequency_engineering" title="Radio-frequency engineering">Radio-frequency</a></li>

<li><a href="Signal_processing" title="Signal processing">Signal processing</a></li>

<li><a href="Telecommunications_engineering" title="Telecommunications engineering">Telecommunications</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Chemical_engineering" title="Chemical engineering">Chemical</a></th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Biochemical_engineering" title="Biochemical engineering">Biochemical/bioprocess</a></li>

<li><a href="Biological_engineering" title="Biological engineering">Biological</a>

<ul><li><a href="Bioresource_engineering" title="Bioresource engineering">Bioresource</a></li>

<li><a href="Genetic_engineering" title="Genetic engineering">Genetic</a></li>

<li><a href="Tissue_engineering" title="Tissue engineering">Tissue</a></li></ul></li>

<li><a href="Chemical_reaction_engineering" title="Chemical reaction engineering">Chemical reaction</a></li>

<li><a href="Electrochemical_engineering" title="Electrochemical engineering">Electrochemical</a></li>

<li><a href="Molecular_engineering" title="Molecular engineering">Molecular</a></li>

<li><a href="Paper_engineering" title="Paper engineering">Paper</a></li>

<li><a href="Petroleum_engineering" title="Petroleum engineering">Petroleum</a></li>

<li><a href="Process_engineering" title="Process engineering">Process</a></li>

<li><a href="Chemical_reaction_engineering" title="Chemical reaction engineering">Reaction</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Materials_science" title="Materials science">Materials</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Biomaterial" title="Biomaterial">Biomaterial</a></li>

<li><a href="Ceramic_engineering" title="Ceramic engineering">Ceramics</a></li>

<li><a href="Corrosion_engineering" title="Corrosion engineering">Corrosion</a></li>

<li><a href="Metallurgy" title="Metallurgy">Metallurgy</a></li>

<li><a href="Molecular_engineering" title="Molecular engineering">Molecular</a></li>

<li><a href="Nanotechnology" title="Nanotechnology">Nanotechnology</a></li>

<li><a href="Polymer_engineering" title="Polymer engineering">Polymers</a></li>

<li><a href="Semiconductor_device" title="Semiconductor device">Semiconductors</a></li>

<li><a href="Surface_engineering" title="Surface engineering">Surfaces</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Systems_engineering_(disambiguation)" class="mw-disambig" title="Systems engineering (disambiguation)">Computer</a></th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<li><a href="Computer_engineering" title="Computer engineering">Computer</a></li>

<li><a href="Cybersecurity_engineering" title="Cybersecurity engineering">Cybersecurity</a></li>

<li><a href="Computer_network_engineering" title="Computer network engineering">Networks</a></li>

<li><a href="Robotics_engineering" title="Robotics engineering">Robotics</a></li>

<li><a href="Software_engineering" title="Software engineering">Software</a></li></ul>

</div></td></tr></tbody></table><div></div></td><td class="noviewer navbox-image" rowspan="5" style="width:1px;padding:0 0 0 2px"><div><span typeof="mw:File"></span></div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Engineering_education" title="Engineering education">Engineering education</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Bachelor_of_Engineering" title="Bachelor of Engineering">Bachelor of Engineering</a></li>

<li><a href="Bachelor_of_Science" title="Bachelor of Science">Bachelor of Science</a></li>

<li><a href="Master's_degree" title="Master's degree">Master's degree</a></li>

<li><a href="Doctorate" title="Doctorate">Doctorate</a></li>

<li><a href="Graduate_certificate" title="Graduate certificate">Graduate certificate</a></li>

<li><a href="Engineer's_degree" title="Engineer's degree">Engineer's degree</a></li>

<li><a href="Regulation_and_licensure_in_engineering" title="Regulation and licensure in engineering">Licensed engineer</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Related topics</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Engineer" title="Engineer">Engineer</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Glossary" title="Glossary">Glossaries</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li>Engineering

<li><a href="Glossary_of_aerospace_engineering" title="Glossary of aerospace engineering">Aerospace engineering</a></li>

<li><a href="Glossary_of_civil_engineering" title="Glossary of civil engineering">Civil engineering</a></li>

<li><a href="Glossary_of_electrical_and_electronics_engineering" title="Glossary of electrical and electronics engineering">Electrical and electronics engineering</a></li>

<li><a href="Glossary_of_mechanical_engineering" title="Glossary of mechanical engineering">Mechanical engineering</a></li>

<li><a href="Glossary_of_structural_engineering" title="Glossary of structural engineering">Structural engineering</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Other</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Agricultural_engineering" title="Agricultural engineering">Agricultural</a></li>

<li><a href="Audio_engineer" title="Audio engineer">Audio</a></li>

<li><a href="Automation_engineering" title="Automation engineering">Automation</a></li>

<li><a href="Biomedical_engineering" title="Biomedical engineering">Biomedical</a>

<ul><li><a href="Bioinformatics" title="Bioinformatics">Bioinformatics</a></li>

<li><a href="Clinical_engineering" title="Clinical engineering">Clinical</a></li>

<li><a href="Health_technology" title="Health technology">Health technology</a></li>

<li><a href="Pharmaceutical_engineering" title="Pharmaceutical engineering">Pharmaceutical</a></li>

<li><a href="Rehabilitation_engineering" title="Rehabilitation engineering">Rehabilitation</a></li></ul></li>

<li><a href="Building_services_engineering" title="Building services engineering">Building services</a>

<li><a href="Design_engineer" title="Design engineer">Design</a></li>

<li><a href="Explosives_engineering" title="Explosives engineering">Explosives</a></li>

<li><a href="Facilities_engineering" title="Facilities engineering">Facilities</a></li>

<li><a href="Forensic_engineering" title="Forensic engineering">Forensic</a></li>

<li><a href="Climate_engineering" class="mw-redirect" title="Climate engineering">Climate</a></li>

<li><a href="Geomatics_engineering" class="mw-redirect" title="Geomatics engineering">Geomatics</a></li>

<li><a href="Engineering_drawing" title="Engineering drawing">Graphics</a></li>

<li><a href="Industrial_engineering" title="Industrial engineering">Industrial</a></li>

<li><a href="Information_engineering" title="Information engineering">Information</a></li>

<li><a href="Instrumentation_engineering" class="mw-redirect" title="Instrumentation engineering">Instrumentation</a>

<ul><li><a href="Instrumentation_and_control_engineering" title="Instrumentation and control engineering">Instrumentation and control</a></li></ul></li>

<li><a href="Logistics_engineering" title="Logistics engineering">Logistics</a></li>

<li><a href="Engineering_management" title="Engineering management">Management</a></li>

<li><a href="Engineering_mathematics" title="Engineering mathematics">Mathematics</a></li>

<li><a href="Mechatronics" title="Mechatronics">Mechatronics</a></li>

<li><a href="Military_engineering" title="Military engineering">Military</a></li>

<li><a href="Nuclear_engineering" title="Nuclear engineering">Nuclear</a></li>

<li><a href="Ontology_engineering" title="Ontology engineering">Ontology</a></li>

<li><a href="Packaging_engineering" title="Packaging engineering">Packaging</a></li>

<li><a href="Engineering_physics" title="Engineering physics">Physics</a></li>

<li><a href="Privacy_engineering" title="Privacy engineering">Privacy</a></li>

<li><a href="Safety_engineering" title="Safety engineering">Safety</a></li>

<li><a href="Security_engineering" title="Security engineering">Security</a></li>

<li><a href="Survey_engineering" class="mw-redirect" title="Survey engineering">Survey</a></li>

<li><a href="Sustainable_engineering" title="Sustainable engineering">Sustainability</a></li>

<li><a href="Systems_engineering" title="Systems engineering">Systems</a></li>

<li><a href="Textile_engineering" class="mw-redirect" title="Textile engineering">Textile</a></li></ul>

</div></td></tr><tr><td class="navbox-abovebelow" colspan="3" style="font-weight:bold;"><div>

<ul><li><span class="noviewer" typeof="mw:File"><span title="Category"></span></span> Category</li>

<li><span class="noviewer" typeof="mw:File"><span title="Commons page"></span></span> <a href="https://commons.wikimedia.org/wiki/Engineering" class="extiw external" title="commons:Engineering">Commons</a></li>

<li><span class="noviewer" typeof="mw:File"><span title="WikiProject"></span></span> Wikiproject</li>

<li><span class="noviewer" typeof="mw:File"></span> <a href="Portal%3AEngineering" title="Portal:Engineering">Portal</a></li></ul>

</div></td></tr></tbody></table></div></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2025-06-05" href="https://en.wikipedia.org/wiki/?title=Data_engineering&oldid=1294149660">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>